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ABSTRACT 
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Executive Summary 



The following document explores the historical issues and current applications 
surrounding the assessment of educational outcomes. Prompted by national, regional, and 
local calls for the evaluation of institutional effectiveness, the authors have attempted to set a 
frame of reference for East Texas State University's response to program evaluation. 

As the promises of universal education have become realities for many Americans 
during the past eighty seven years, educational institutions have come to realize that providing 
programs is no assurance that these programs are operating effectively. As a result, systematic 
and integrative assessment has come io be acknowledged as a safeguard to the educational 
rights of all citizens. Responding to this paradigm shift toward greater student-centerdness, the 
Southern Association of Schools and Colleges and the Texas Coordinating Board have adopted 
language indicating that assessment practices are central to ascertaining the effectiveness of 
educational institutions in serving their constituents. 

As emphasized throughout the present working paper, the nature of program evaluation 
is congruent with the historical values of East Texas State University. A student oriented 
curriculum designed to serve the needs of our region, an investigative faculty devoted to 
evaluating and promoting the values of its programs, an informed administration committed to 
providing a forum for learning - these have been and are the values of our University. As such, 
faculty, students, and administrators should realize that our institution has long been sensitive 
to the evaluation of its educational objectives. 

Therefore, the authors have speculated systematically regarding a model by which our 
institution may preserve its place as an educational leader. Through the coordination of 
nationally and locally designed assessment practices, our institution may fully come to realize 
that environment in which faculty come together from across the disciplines to formulate, 
execute, analyze, and refine the aims and practices of the educational community. 

Challenging times demand creative and timely responses. We would hope that the 
following working paper, suggestive and argumentative in its nature, will stimulate the 
educational community here at East Texas State University to discussion of creative and timely 
asse» *ment of our University's educational efforts. 
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1. Introduction: The Aim of this Working Paper 



In its 1984 Criteria for Accreditation, the Southern Association of Colleges and Schools 
called for its member institutions to provide systematic evaluation of educational programs: "The 
level of institutional quality depends not only on an institution's educational processes and 
resources but also on the institution's successful use of these processes and resources to achieve 
established goals, institutions have an obligation to all constituents to evaluate effectiveness and to 
use the results in a broad-based continuous planning and evaluation process M (10). 

In order to respond to the Southern Association's call for proof of institutional effectiveness, 
the authors of the present wori< have tried to articulate issues and suggest methods by which East 
Texas State University may actively establish systematic and continuous programs evaluation. The 
authors wish to stress that their ideas are intended to serve as a stimulus rather than a prescription 
for discussion. Their aim has been to provide information and prompt discussion so that the 
E.T.S.U. community may develop programs which address the Southern Association's call for 
evaluation of institutional effectiveness. 



2. Historical Notes: Educational Evaluation in America 



Although education in America during the seventeenth and eighteenth centuries was 
classically oriented and elitist, by the 1830s social policy, striving to fulfill the promises of 
democracy, began to press for free universal elementary schooling. By the turn of the century this 
goal had been achieved. Once achieved, however, an ambitious nation realized that advanced 
training of its citizens was essential to a strong social and economic structure; therefore, demands 
for secondary education became greater. Still later in the century, the same themes again came into 
play as high school diplomas became merely a slep along the way to a college degree. 

Concurrent with the expansion of the goals and costs of American education was an 
increased emphasis on accountability. As the promises for a true public schooling have become 
realized, mere illustration that services are provided is no longer adequate. Educational institutions 
have come to realize that serving their students and knowing their capabilities are synonymous 
activities. In present day America it is the uncommon educational endeavor which does not provide 
for assessment of that endeavor. Simply put, evaluation has become a part of educational America. 

These historical developments are not to suggest that assessment is a simplistic issue. 
Assessment must irdex complex programmatic and environmental factors. Economically, for 
instance, from World War II until the 1980s our public universities enjoyed a period of tremendous 
growth. The National Defense Education Act of 1958,the National Science Foundation, the Ford 
Foundation grants of 1967 - all are indicative of historically unprecedented financial support for the 
American college curriculum, Yet in 1982-1983, due to complex economic constraints effecting 
every segment of all American institutions, the total financial appropriation by state governments for 
higher education was increased by only 6% over the previous year. This level of support was the 
lowest percentage of increased funding in twenty years. 

Here in Texas, educators have witnessed a 9% decrease in state support for higher 
education, as the map below indicates: 



HOW STATE SUPPORT HAS CHANGED IN 2 YEARS 




^ * 25% or mort 
P ♦ IS* to ♦ *<% 



(Chronicle 1) 
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In fact, Texas presently ranks 49th in the nation in terms of percentage increases in funding for 
education (Chronicle 21). 
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The issue of assessment, then, is made more complex by economic constraints. How do 
we begin to undertake assessment, educators justifiably ask, in a period of economic crisis? 
Effective assessment requires funding and time, both of which are in short supply throughout 
educational America. A college instructor's professional life is divided among scholarship, 
teaching, and service; now another factor - assessment - must be addressed, still another 
demand for which there appear to be few rewards. 

Clearly, there is no simplistic solution to this very real, very immediate dilemma. Perhaps 
one useful path to investigate, though, might be to adopt a unified attitude toward our professional 
lives in which scholarship, teaching, service ; and assessment come to be viewed not as mutually 
exclusive activities but as complementary intellectual activities which, taken together, give our lives 
authenticity and substance. The diagram below suggests such a vision: 

Student-Centerud Aim of East Texas State University 




Unified in an atmosphere of studeni-centerdness, educational activities take on new vitality and 
meaning. 

Perhaps examples taken from among our faculty would serve here. Through the Department 
of Literature and Languages, the North East Texas Writing Project promotes state-of-the-art 
strategies in the teaching of writing for public school teachers in Region 10 and then empirically 
evaluates the work of students enrolled in the classes of these teachers. Also of that Department. 
Drs. Jon Jonz and Fred Tarpley work extensively with area teachers in the instruction of English as a 
second language and have reported to national audiences the results of their work; and Dr. Ann 
Moseley has published on the curriculum of our University's Communication Skills Center. In the 
Department of Elementary Education, Drs. Michael Sampson and Joseph Vaughn work with 
teachers to develop effective methods in the teaching of reading. In the Department of Psychology, 
study programs developed by Dr. Bernadette Gadzella have been adopted by universities 
nationwide. Dr. Dean Ginther, also of the Department of Psychology, has published extensively on 
the use of personal computers in the classroom. Dr. Fred Blohm of the Department of Health, 
Physical Recreation, and Education has written on his investigation of physical education programs 
in Texas. In the Department of Elementary Education, Dr. Rosemarie Kolstad has been recognized 
internationally for her work in curriculum development, and Dr.Reba Hudson of the College of 
Education continues to assimilate and interpret information on the performance of our teacher 
education students in locally developed and nationally administered assessments. 

On an interdisciplinary level, faculty from across the curriculum join in developing and 
reporting their efforts in student-centered activities. Our University Studies Program is informed by 
recent advances in general education philosophy and promises to be a national model, and our 
New Center for Learning has a long histcy of nationally reported program development. Under the 
coordination of Dr. Kolstad, Drs. James Kushland in the Department of Biologv, Williard McDaniel 
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and Dev Chopra of the Department of Physics, and George Nixon of the Department of Chemistry 
have come together to contribute toward the development of a curriculum for science teachers. 
Too, Drs. William Richardson of the Department of Elementary Education, Kenneth Sheppard of the 
Department of Earth Sciences, Stephen Razniak of the Chemistry Department, and Dr. William 
Ogden of the Department of Secondary and Higher Education have developed programs in science 
education and roported their research nationally. In the Department of Special Education, Dr. 
Maximino Plata, along with colleagues, has received both regional and national recognition for work 
on the Junior Level Essay, our mandated rising junior examination. Of course, these examples are 
illustrative and not inclusive. 

Such meaning and vitality expressed in these individuals and their research emphatically 
addresses the Southern Associations call for assessment of institutional effectiveness.lt is a unified 
effort toward an educational community for which we strive here at East Texas State University. It is 
student-centerdness for which our institution is known, and it is that student-centerdness which we 
must preserve and protect. 

In order to investigate more fully the nature of Section III, we will now turn to an analysis of 
that part of the Criteria . 



8 



5 



3. Section III of the Criteria : An Interpretation 



So that our interpretation of Section III may be advanced, we have reproduced below the 
entire text under consideration: 



The quality of education provided by member institutions is the primary consideration in the decision to confer or 
reaffirm accreditation. The evaluation of educational quality is a difficult task requiring careful analysis and professional 
judgment. Traditionally, accreditation has focused attention almost exclusively upon institutional resources and processes. 
It has usually been assumed that, if an institution has certain resources and uses certain processes, effective education will 
occur. A comprehensive approach to accreditation, however, takes into account not cnly the resources and processes of 
education (such as faculty and student qualifications, physical plant, fiscal resources and other elements addressed in the 
Criteria) but also the valuation of the results of education and plans ior the impiovement of the institution's programs. The 
level of Institutional quality depends not only on an institution's educational processes and resources but also on the 
institution's successful use of those processes and resources to achieve established goals. Institutions have an obligation 
to all constuents to evaluate effectiveness and to use the results in a broad-based, continuous planning and evaluation 
program. 



To focus on the effectiveness of the educational program, the institution must establish 
adequate procedures for planning and evaluation. The Institution must define Its expected educational 
results and describe how the achievement of these results will be ascertained. Although no specific 
format for this planning and evaluation processes is prescribed.an effective process should Include: 



In addition to establishing procedures for evaluating the extent to which their educational goals 
are being achieved, Institutions should ascertain periodically the change in the academic achievement of 
their students. Procedures used to evaluate instructional programs may include: peer evaluation of 
educational programs; structured Interviews with students and graduates: changes In students' values 
as measured by standard Instruments of sefrreported behavior patterns; pre- and post-testing of 
students; surveys of recent graduates; surveys of employers of graduates; student scores on 
standardized examinations or locally constructed examinations; performance of graduates in graduate 
school; performance of graduates of professional programs on licensure examinations; or, the placement 
of graduates of occupational programs in positions related to their fields of preparation. 

Institutions with research or public service missions must develop and implement appropriate 
procedures for evaluating their effectiveness in these areas. 



Section III 
Institutional Effectiveness 



3.1 Planning and Evaluation 



1. 
2. 
3. 
4. 



broad-based invoivement of faculty and administration; 

the establishment of a clearly defined purpose appropriate to collegiate education; 

the formulation of educational goals consistent with the institution's purpose; 

the development of procedures for evaluating the extent to which these educational goals are 

being achieved; and 

the use of the results of these evaluations to improve institutional effectiveness. 



5. 
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The appropriateness of any evaluation procedure depends upon the nature of the institution 
snri the institution's goals for instruction, research and public service. The Commission on Colleges 
prescribes no set of procedures for use by an institution and recognizes that an effective program to 
evaluate institutional effectiveness will usually require the use of a variety of procedures. 

3.2 Institutional Research 

Because institutional research can provide significant information of all phas>ss of a college or 
university program, it is an essential element in planning and evaluating the institution's success in 
carrying out its purpose. The nature of the Institutional research function depends on the size and 
complexity of the Institution and may vary from part-time operation to an office staffed by several persons. 
AH Institutions, however, must engage in continous study, analysis and appraisal of their purposes, 
policies, procedures and programs. Institutions should assign administrative responsibility for carrying 
our institutional research. 

(Criteria 10-11) 

As stressed in the previous section, the trend in education which emphasizes accountability 
for programs is present throughout Section III. The two introductory paragraphs argue that 
educational quality, the "primary consideration" in accreditation decisions, is tied to demonstrable 
measurement of that quality. Such assessment, the Southern Association implicitly argues, is not to 
be taken as a superficial measurement executed over a finite period. The assessment must be 
continuous, it must be broad based, and it must be tied to the goals of programs. 

Subsection 3.1 further explains the planning and evaluation process by emphasizing the 
necessity of heightened focus: an articulation of the mission statements of the institution and its 
programs. Formally composed and unified in their goals across the curriculum, those mission 
statements will control the domain of the assessment as well as the selection of the instruments 



Subsection 3.1 further identifies ten procedures to evaluate instructional programs: 



1. 


Peer evaluation of educational programs, 


2. 


Structured interviews with students and graduates, 


3. 


Changes in students' values as measured by standard instruments or self-reporting 




behavior patterns, 


4. 


Pre- and post-testing of students, 


5. 


Surveys of recent graduates, 


6. 


Surveys of employers of graduates, 


7. 


Student scores on standardized examinations or locally constructed examinations, 


8. 


Performance of graduates in graduate school, 


9. 


Performance of graduates of professional programs on licensure examinationt,and 


10. 


Placement of graduates of occupational programs in positions related to their fields 




of preparation. 



Significantly, the Southern Association is not asking solely for positivistic pre-instruction/post- 
instruction design. Although pre- and post-testing of students is indeed part of the assessment 
process, it is not the only part. Testing, interviews, surveys, and longitudinal tracking of students are 
all part of a comprehensive program of evaluation. And again in Subsection 3.1. the Southern 
Association emphasizes the necessity of articulated mission statements defining "the nature of the 
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4. Our Mission Statement and Our Alms at E.T.S.U.: An Analysis 

Below is E.T.S.U.'s mission statement: 

East Texas State University is a major center of learning for the Northeast Texas region. The 
university is responsible for teaching, scholarly research and creative activities, and public service. 

Through Instructional programs of the highest quality the university develops the skills and 
knowledge necessary for professional success, personal growth, and responsible citizenship. Through 
both formal and informal activities, students have the opportunity to acquire knowledge; learn values and 
potentialities; and develop an enduring commitment to learning. 

Through support of research, scholarship and creative endeavors the university helps to 
advance human achievement and stimulate the intellectual growth of faculty, students, and all others who 
enjoy the products of these activities. 

Through the provision of information, facilities, research expertise, public education, and 
cultural activities, the university serves its constituency. 

Upon analysis, we find our University is conceived in comprehensive terms: scholarship and 
teaching inform each other. In the context of ths approach, students begin to view themselves as 
holistic learners. They are encouraged toward "professional success, personal growth and 
responsible citizenship." Ultimately, students develop what John Henry Newman called a habit of 
learning. 

In order to foster this habit, our University's mission statement suggests that programs not 
be defined merely as those components of our system through which degrees are granted. 
Instead, a program is better defined as those activities of E.T.S.U. which contribute to the 
advancement of our students' professional success, personal growth, and responsible citizenship. 
In light of this broader and more informed view of programs, ventures such as University Studies, 
Net Ole, the New Center for Learning, and the Center for Policy Studies take their place along with 
those programs leading to degrees at the bachelor's, master's, and doctor's levels. 

As is the case with our methods of assessment discussed in the previous chapter, E.T.S.U. 
presently has well developed and successful programs at all levels. However, just as we must 
collectively evaluate our existing programs of assessment, so too must we delineate these existing 
programs themselves and decide which should receive priority in terms of assessment. Since 
steps toward giving priority to programs have already been undertaken under the supervision of the 
Associate Vice President for Academic Programs and Services and individual department heads, 
this process is already in progress. 

With these issues of assessment and mission in mind, we may now offer some general 
paradigms for assessment and adapt these paradigms to our University's present needs. 
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5. Some Paradigms for Assessment 



In the assessment of educational outcomes there are essential intelectual patterns - 
paradigms - which control the conditions of the assessment. Structurally, we might say that all 
assessments are controlled by paradigms centered around the concepts of aim, domain, and 
instrument. 

The aim of an educational assessment centers around the basic beliefs of a program. That 
is, we investigate the commitments of our programs and describe how these commitments might 
be implemented. 

The domain of an educational assessment is based on distinctions between whether we 
w : si; to gather cognitive or affective infomation about our students (Ewell 13). In focusing on 
cognitive outcomes, we concentrate on gains in student knowledge. We seek, for instance, to 
measure increased knowledge of British literature or the ability to think critically. In focusing of 
affective outcomes, we cor intrate on gains in student values. V/e seek to measure changes in 
attitudes, tolerance, or prio» <y of judgments. 

Instruments, those tests or surveys selected to serve the aim and domain of the evaluation, 
a.e chosen on the basis of whether they are nationally normed or locally developed. With nationally 
normed tests, we are able to see how our students perform in relationship to students across the 
nation. We therefore acquire a sense of broad-based performance, a sense which breaks down 
regional isolationism. Locally developed tests, on the other hand, afford those designing the 
assessment complete control of fr<* nature and type of measurement. As a result, locally developed 
tests allow an enhanced sense of community among students, instructors, and administrators. As 
we come to understand the local characteristics of our students, programs, and faculty, we develop 
a cogent sense of the uniqueness of the institution, a uniqueness affording a sensitivity toward the 
needs of special populations such as handicapped and minority students. (We will turn to soma 
guidelines for the adoption of nationally normed and locally developed tests in Section 7.) 

With the concepts of aim, domain, and instrument in mind, we may formulate the archetypal 
model below for educational outcomes: 



Universal Model for Assessment 
of Educational Outcomes 



Aim & Domain 
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As the model indicates, aim is the controlling factor in the assessment.determining not only 
the domain and instrument but also our frame of reference in assessing pre- and post-enrollment 
populations. 

Based on this model, we may then formulate the diagram below for assessment of our 
undergraduate programs: 

ETSU Model for Undergraduate Assessment 



Aim & Domain 




Here, we would employ assessment instruments at both the pre-enrollment and post-enrollment 
periods in order to better understand the expectations of and results from our efforts. Additionally, 
assessment at both the freshman and senior levels allows investigation of gams before and after 
comprehensive educations experiences. 

To more explicitly define the nature of these assessments, we have indicated in the diagram 
below that testing and survey instruments would provide both cognitive and affective information: 



Defined ETSU Model for Undergraduate Assessment 



Aim & Domain 




Here, instruments have been defined as those which test and those which assess. Note that pre- 
testing has been suggested in the freshman year and post-testing in the senior year and that 
surveys may oe undertaken before enrollment, during the freshman and senior years, and after 
enrollment. Such a pattern of testing and surveys will also limit the overall data collection process 
while yielding valuable information on our students. 
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Too, we may formulate a fundamental plan for graduate assessment: 
ETSU Model for Graduate Assessment 



Aim & Domain 




As the diagram above indicates, v/e must assess our students at both the master's and post- 
master's levels; as well, we may gain valuable information about the hopes and gains of our 
students in examining them before and after enrollment in our graduate programs. 

Additionally, we may define this model be'ow: 

Defined ETSU Model for Graduate Assessment 



Aim & Domain 
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Here, we indicate that at E.T.S.U. testing might occur at both the master's and post-master's levels 
and that surveys may be administered at all four levels of student progress. 

Models of assessments such as those offered above have already proven effective in 
nationally recognized programs at the University of Tennessee at Knoxville (Banta), Northeast 
Missouri State University (McClain and Krueger) and Alvemo College (Mentkowski and Loacker). In 
each case, the programs have proven successful, especially in those states such as Tennessee 
whose Higher Education Commission now applies academic performance criteria in funding of state 
colleges and universities. Research tells us that we have every reason to believe that the 
assessment models offered above will prove of value. 

With these models in mind, we now present a case study in which an articulated model is 
offered for our University Studies program. 
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6. A Case Study: Assessing University Studies 

Since our program of University Studies has recently been reformulated and is therefore 
perhaps the single program on campus most familiar to all faculty, it presents itself as a logical 
candidate for speculation about possible assessment procedures. 

As discussed in the previous chapter, the aim of a particular program must be the place 
from which all further investigation develops. In order to address the assessment of University 
Studies at E.T.S.U., we therefore must begin with the philosophy of general education given below: 

General education is an essential element in the educational process that results in 
professional success, personal growth, and responsible citizenship. In this process, both formal and 
informal, students have the opportunity to acquire knowledge, learn reasoning and communication skills, 
examine their goals, values and potentialities, and develop an enduring commitment to learning. 

The general education program at ETSU embodies the characteristics and purposes of the 
university. As a university with both professional and liberal arts programs, ETSU recognizes general 
education as a necessary component in the development of all students. 

General education is not only compatible with specialization, but is the context in which 
specialization occurs. General education is a coherent program that places emrhasis r common 
learning of a shared body of knowledge, the development of intellectual skills, and an appreciation and 
toleration of diverse attitudes and values. 

Specifically the general education program stresses breadth and provides a person with 
opportunities to perceive the integration of knowledge; involves the study of basic liberal arts and 
sciences; encourages understanding of our heritage as well as respect for other peoples and cultures; 
develops mastery of linguistic, analytical, and computational skills that are necessary for lifelong learning; 
and fosters development of such personal qualities as appropriate acceptance of ambiguity, empathy 
and acceptance of others, and an expanded understanding of self. 

(Report to Or. Morris 4) 

In selecting survey and testing instruments, we must realize that the oomain of general 
studies is properly both affective and cognitive. The emphasis of University Studies is toward 
fostering throughout the undergraduate curriculum "a shared body of knowledge, the development 
of intellectual skills, and an appreciation and tolerance of diverse attitudes and values." Taken 
together, these aims suggest the changes in values most readily assessed by affective instruments. 
Of course, this is not to suggest that gains in specific kinds of knowledge are unimportant but rather 
that University Studies seeks to establish a meaningful context in which specialization will occur. 
Assessment of both the affective and cognitive domain are thus important to the success of our 
University Studies program. 
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Based on Lawrence Kohlberg's developmental moral stages (xxix), James R. Rest's 
Defining Issues Test (DIT) would perhaps serve as an identifier of affective development in students 
enrolled in University Studies courses. As of 1986, over 500 studies with the DIT have provided 
significant information on those factors which prompt moral development as measured by that 
instrument, and it would be of use to the University Studies program to discover whether or not our 
program is addressing its aims of providing a ground for students to develop their values and 
potentialities. In the cognitive domain, the American College Testing (ACT) program's College 
Outcomes Measures Project (COMP) holds promise for a meaningful assessment of general 
studies. Over 300 colleges have used the COMP instrument, and formal studies of its effectiveness 
have also been conducted. Addressing six fundamental outcome areas - communication, problem 
solving, value classification, functioning within social institutions, using science and technology, 
and using the arts (Forrest 11-12) - the COMP seems tailored for our University Studies' emphasis 
on acquisition of knowledge, reasoning skills, and communication skills. 

In order to have a locally developed instrument designed to assess our students' writing 
skills, our Junior Level Essay (JLE) has served us well in the past in helping us describe our 
students' ability to compose. Over 3,000 of our students have been effectively assessed in their 
writing skills through this instrument (Plata, Elliot, Zelhart), and there is every indication that it is 
sensitive to the skills cf our unique student population here in Commerce. 

In turning from testing to surveying, we believe that the sort of questionnaires offered by the 
College Board's Multiple Assessment Programs and Services (MAPS) would yield significant 
information about our students' opinions of University Studies. If admin ! *tered at both the freshman 
and senior levels, we could trace our students' reactions to general studies here at E.T.S.U. and 
compare their reactions to other general studies programs across the country through the College 
Board network. 

Therefore, an articulated plan for assessment of our University Studies program might 
resemble the diagram below: 



Articulated ETSU University Studies Plan 



Aim & Domain 
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Since University Studies is the one program at E.T.S.U. influencing the education of every 
student, the mt:1iple assessment measures such the ones described above would tell us a great 
deal about the development and progress of our students while addressing the Southern 
Association's call for evaluation of institutional effectiveness in Section III. 

To more clearly understand how all programs at E.T.S.U. may address issues of 
assessment, we will now suggest guidelines for addressing issues of aim, domain, and instrument 
selection. 
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7. Developing Assessments of Individual Programs: 
Considerations for E.T.S.U. Faculty 



In Sections 5 and 6 we focused on explaining some rather general systems for assessment 
and applying these to our situation here at East Texas State University. The key to our University's 
success in responding to Section III of the Criteria , however, rests on our faculty's ability to develop 
its own assessment methods for individual programs. In this section, therefore, concepts of aim, 
domain, and instrument will be more fully examined as they apply to E.T.S.U. program assessment. 

To begin, we would argue that the aims of every program in our University must reflect the 
aims of E.T.S.U.'s mission statement. Specifically, we recommend that the mission statements, the 
aims of all programs, include emphasis on the following: 

the program's ability to enhance education in the Northeast region of Texas; 
the program's ability to unify teaching, scholarly research and creative activities, and 
public service; 

the program's emphasis on professional success, personal growth, and responsible 
citizenship; 

the program's emphasis on diverse types of educational experiences for students so that 
their intellectual and moral growth may be enhanced; and 

the program's emphasis on how its specific knowledge contributes to an individual's 
holistic view of success. 

Essentially, every program should be considered in terms of its overall student- 
centerdness. The stated domain of each program will properly focus on a combination on the 
cognitive and affective realms. That is, an affective program will focus on: 

gains in student knowledge, and 
gains in student values. 

A sensitive combination of these domains in a program description will strive to show how increases 
in specific knowledge contribute to an attitude of understanding and comprehensiveness in our 
students. 

Based on aim and domain, instruments may be conceived in terms of standardized and 
locally developed assessments. Standardized assessments, defined as those "commercially 
prepared by experts in measurement and subject matter"(Mehrens and Lehmann 7), are helpful in 
allowing their nsers to compare students' performance to other examinees who have taken the 
assessment. Locally developed measures, on the other hand, allow their developers greater control 
of the content of the assessment and of subtlety in its interpretation. 

Of course, both standardized and localiy developed assessments may be used to mea? jre 
the cognitive and effective domains. In selecting a cognitive measure of writing ability for a general 
studies program, for instance, a faculty might use thp standardized Test of English Composition 
developed by the College Board's Multiple Assessment Programs and Services. Similarly, a 
standardized affective survey might include the Terminal and Instructional v ue Scales by Rokeach 
or the Self Defining Issues Test by Rest. 
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In investigating locally developed cognitive assessments, a faculty might develop an 
holistically scored essay examination such as our own Junior Level Essay, in the affective domain, 
faculty might develop a form of the well-known Likert scale to measure the attitudes of students. 

In order to understand more clearly our own goals in deciding to use standardized or locally 
developed assessments, the following guidelines might prove helpful: 



Guidelines for Selecting a Standardized Test 

Is the aim of the assessment suited to the domain and instrument under 
consideration? 

Is the purpose and recommended use of the instrument explicitly described in the 
test manual provided by the publisher? 

What degree of success has the instrument yielded for populations similar to those 
under consideration? 

How recently has the instrument been developed? That is, has the instruments 
design been informed by recent pertinent research? 

Does the instrument follow the Standards for Educational and Psychological Testing 
set forth by the American Educational Research Association, the American 
Psychological Association, and the National Council on Measurement in Education? 
Is the cost of the instrument acceptable? 

(Derived from Mehrens and Lehmann 34-35) 

Guidelines for Developing a Local Test 

Have the issues in the following basic test development procedure been addressed? 

1. Decide central issues: 

purpose 

characteristics of population 
use of test 

implications of use of test 
cost 

type of assessment 

selection of test development committee 
selection of item writers 
time line 

2. Preparing to write the test 

(Specifications defined by test development committee) 
purpose 

subject to be tested 
population 
use of test 
implications of test 
timing 

number and types of items 
statistical specifications 
anticipated disadvantages 
anticipated advantages 



1. 
2. 
3. 
4. 
5. 

S. 
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3. Writing the test 

item writers construct test 
item writers review test 

4. Pretesting of items 

analysis of validity,reliability, and bias of pretest 

5. Preparing final form of test 

printing 
copying 

6. Preparing population to be tested 

public disclosure of specifications through 
correspondence and media 
availability of test development committee 

7. Administering the test 

instructions for administrators 
security of test 

8. Scoring the test 

methods dependent on type of assessment 

9. Analyzing test results 

validity, reliability, and bias 

10. Documenting results of test 

1 1 . Interpreting test results :o public 

corresponded ,e and media 

12. Observing how test influences instructor and student behavior 

13. Analysis of procedure and results by test development committee 

14. Presenting and publishing test development procedures and results 

correspondence 

workshops 

conferences 

articles 

monographs 

books 

(Derived from Alloway and Conlan 2) 

Whatever our decision in selecting an instrument, we should keep in mind a few central ideas when 
selecting developing our evaluative procedures. 

First, we need to remember that the development of all assessment measures is radically 
historical. The specific aims of a particular program must be discussed, analyzed, and expressed 
before a particular type of instrument may be either selected or developed. Far too often there are 
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tendencies for new programs of assessment to adopt, unexamined, either a standardized test or 
another institution's locally developed test, thus creating misunderstandings among administrators, 
instructors, and students. The process of a specific group of instructors assembling to look 
intelligently and deeply at their own programs is critical to the success of any assessment of that 
program. Individuals must come together in a spirit of inquiry to investigate specific phenomenon 
(in our case, specific programs) and, through a process of dialogue, achieve concensus. 

Second, we hold that in a community wide tssessment, the entire community must 
participate in the assessment's development. Program assessments must be established in a spirit 
of freedom and openness, and we must ensure that the academic community affected by our 
programs share in the opportunity of developing the assessment of that program. All involved in a 
specific program must participate in shaping that program's assumptions, practices, and 
assessment procedures. 

Third, we hold thai the connections between assessment and classroom teaching must 
always and everywhere be present. The educational impulse of our profession-to question and 
clarify-should thus guide us as we begin to investigate the relationships between instruction and 
assessment. 

Finally, we must remember that assessment is a process that must proceed in a dialectic, 
recursive pattern. As we establish our tests and surveys, we must therefore not suggest that 
assessment of programs takes place on a one-time basis over a finite period. Ultimately, the 
process of assessment must reflect the process of program development itself. 

With these considerations in mind, we may now turn to the expression of programs of 
assessment through what may be called the Assessment Document. 



ERIC 



23 



20 



8. Developing the Assessment Document 



After each program at E.T.S.U. is identified, those responsible for that program will begin to 
discuss how it may be most effectively evaluated. When the best method of evaluation is decided 
upon, the program developers will compose a report which we have termed the Assessment 
Document. The purpose of this Document is to define for the rest of the university community an 
integrative set of descriptions and plans which delineate the background, aim, domain, and 
assessment instruments of the program. Taken together, these Documents will provide a unified 
picture of programs and assessment practices. 

To indicate the shape that the Assessment Document might take, we have drafted the 
following mock paper which discusses the assessment of our own University Studies program: 



In. October of 1982. the Vice President of Academic Affairs appointed a University General 
Studies Committee and charged it to Investigate a plan for action in the implementation of a general 
studies curriculum at East Texas State University. Workshops, surveys, and position papers followed 
over the next three years until the spring of 1985 when a draft of a proposal for a general studies 
curriculum was circulated among the faculty. After the proposal was revised and suggestions for general 
studies courses were submitted from disciplines across the curriculum, a program of University Studies 
was completed during the spring of 1987. 



University studies is an essential element in the educational process that results in 
professional success, personal growth, and responsible citizenship. In this process, both formal and 
Informal, stuocnts have the opportunity to acquire knowledge, learn reasoning and communication skills, 
examine their goals, valnes, and potentialities, and develop an enduring commitment to learning. 

The University Studies program at E.T.S.U. embodies the characteristics and purposes of the 
University. With both professional and liberal arts programs. E.T.S.U. recognizes general education as a 
necessary component In the development of all students. The general education which takes place in the 
University Studies program is therefore not only compatible with specialization but is the context in which 
specialization occurs. 

University Studies is a coherent program that places emphasis on common learning of a 
shared jody of knowledge, the development of intellectual skills, and an appreciation and toleration of 
diverse attitudes and values. 

Specifically, the University Studies program! 

stresses breadth and provides a person with opportunities to perceive the 

Integration of knowledge; 

involves the study of liberal arts and sciences; 



The Assessment of University Studies: A Plan 



Background: 



Aim; 
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encourages understanding of our heritage as well ar respect fur other people's 
cultures; 

develops mastery of linguistic, analytical, and computational skills that are necessary 
for lifelong learning; and 

fosters development of such personal qualities as appropriate acceptance of 
ambiguity, empathy and acceptance of others, and an expanded understanding of 
self. 

Domain: 

University studies strives to enhance student knowledge and values. The program therefore 
promotes both cognitive and affective development In order to foster a complete sense of Intellectual 
and moral development. 

Instruments: 

So that student knowledge and values may be evaluated, assessment Instalments will focus 
on both testing and surveying. The American College Testing program's College Outcomes 
Measurement Project is currently proving quite successful in helping faculty at institutions such as the 
University of Tennessee at Knoxvllle to assess their general studies program. This standardized test 
would be given at both the freshman and senior year so that gains In student knowledge and reasoning 
may be addressed. 

A second standardized test, the Defining Issues Test, serves to assess the affective domain. 
The Defining Issues Test is currently used by such model general studies programs as Arvemo College 
in Milwaukee. To assess the growth of our students, this test would be given along with the College 
Outcome Measurement Project test In both the freshman and senior years. 

A locally developed test, our Junior Level Essay is the present form of a test of writing ability 
begun in 1950 at our University. With revision of the instrument in 1984, the test has served E.T.S.U. 
students weM in helping faculty to assess the strengths and weaknesses of the examinees' writing ability. 
The Junior Level Essay wl!! be given at the junior year so that students whose skills are shown to be 
deficient may be placed in coursework so that their skills may be strengthened by the end of the senior 
year. 

Finally, our students may be surveyed in their attitudes toward academic and career Interests 
through the Comparative Interest Index of the American College Testing program. The Index, yielding 
scores on eleven scales derived from academic discvlines, would tell us a good deal about the 
developing Interests of our students. To tailor this survey to our program, we may insert twenty-five items 
of our own design. So that we may track our students' shifting attitudes, we will administer this survey 
both in our freshman and senior years. 

Schedule: 

Assessment procedures will commence In the fall of 1988. Since our University Studies 
program will be fully In place by that time, we will begi/i our assessments next fall and continue them for a 
four year period through 1992. Throughout this process, information will be gathered, Interpreted, and 
communicated to University Studies faculty so that our on-going program of assessment may Influence 
curriculum. 
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Summary: 

The assessment procedure for University Studies at E.T.S.U. is suggested in the diagram 

below: 

Articulated ETSU University Studies Plan 



Aim I Oomtln 




This comprehensive assessment program will clearly comply with the Southern Association's call for 
assessment of educational outcomes. As significant, this assessment program wiH provide our faculty 
with meaningful information about the effectiveness of their student-centered efforts. 

The sample Assessment Document above illustrates the sort of briefly documented 
narrative report which, when completed by the faculty responsible for the program, will provide unity 
for a univsrsiiy wide evaluation of institutional effectiveness. Specifically, the aim of the Assessment 
Document points to the integrative aims of the program, aims reflected in both the cognitive and 
affective domains. The instruments for the assessment employ both standardized and locally 
developed measures, thus affording both national comparisons and unique information about our 
specific situation. 

So far in this wooing paper we have tried to provide an approach to the evaluation of 
educational outcomes. To this end, we have offered historical background, archetypes for 
assessment, and interpretation and application of these archetypes. With the contours of our 
assessment situation defined, in the next section we will suggest a time line for E.T.S.U.'s response 
to Section III. 
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9. Establishing a Time Line 



By September 1, 1988, all post-secondary institutions in the Southern Association of 
Colleges and Schools have been required to file a detailed report defining plans to meet Section III 
requirements. To meel that deadline, at this writing only twelve months away, we offer the following 
time line: 

Major Phases 

9/1/87-9/1/88 

Phase 1 
(Months 1*3) 

Identify and define focus for all program aims. Establish central 
authority for coordination for the project. Identify institutional 
researchers. 

Phase 2 
(Months 4-5) 

Call for, review, and revise written aims of programs to create 
institutional unity. 

Phase 3 
(Months 6-8} 

Meet with individual program developers to discuss preparation of 
Assessment Documents. 

Phase 4 
(Month 9) 

Call for Assessment Document from individual programs. 

Phase 5 
(Months 10-11) 

Review Assessment Documents, revise if necessary, and fcmulate 
long term plans for assessment of institutional effectiveness. 

Phase 6 

(Months 12 and following) 

Local, regional, and national dissemination of information. 
Commence assessments. 
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A plan of action such as this will allow our University community to have in our possession a 
comprehensive plan of assessment within a one year period. In following such a plan, we will have 
identified and established 

a central authority for assessment, 

a statement of the aims of every program on campus, and 

a plan for evaluation of these programs. 

Once our frame of reference is set, we have but to unify our existing assessment efforts and begin 
to develop and implement those instruments new to us. As the data from our assessments is 
interpreted, we may begin to shape and refine our programs so that their aims may more explicitly 
meet the needs of our students. 

Once our assessment program begins to come to fruition, we will be in a stronger position 
to meet the requirements of both the Texas Coordinating Board and the Southern Association. In its 
Regular Session the 70th Texas Legislature adopted and the governor signed into law many of the 
major recommendations ot the Select Commission on Higher Education. Among thai legislation is 
House Bill 2181 which stipulates ihat the Coordinating Rr^xl will allocate "incentive funding, as a 
percentage of base funding, among institutions to reward those that have achieved goals set by the 
Board in such areas as minority recruitment, graduation rates, meeting planning goals, and energy 
conservation, among others" (Temple 2). Additionally, the Board will also allocate "special initiative 
funding among institutions to promote academic excellence" (Temple 2). Clearly, the language of 
this legislation suggests that leadership in the evaluation of educational outcomes will be 
rewarded. Regarding the Southern Association, E.T.S.U. will be undergoing a mid-term evaluation 
next year. Five years from that date our institution "ill be seeking reaccreditation. By that time, we 
must able to provide the Association with exact ampirical information on the success of our 
programs. 

In essence, the assessment of educational outcomes in an idea whose time has come. 
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10. Conclusions: Assessment of Educational Programs at E.T.S.U. 



We ended the last section by arguing that we ought to begin systematic, on-going program 
evaluation because the political/legal structure of our educational system demands it. Yet we do not 
want to leave the impression that this situation exists because empiricism has somehow gotten out 
of hand and we are being victimized by the technological elements of modern society. Instead, we 
want to conclude this wording paper by arguing that education in America is currently undergoing a 
paradigm shift and the resulting current political/legal structure is but a reflection of that shift. 

In his now classic The Structure of Scientific Revolutions . Thomas S. Kuhn posited that 
intellectual systems - the paradigms of a profession - shift when the existing conceptions of that 
profession reveal shortcomings. In reference to science, Kuhn writes, M when the profession can no 
longer evade anomalies that subvert the existing tradition of scientific practice - then begin the 
extraordinary investigations that lead the profession at last to a new set of commitments, a new 
basis for the practice of science." (6) 

Such paradigm shifts have been noted over the past decade in professions such as English 
(Hairston), psychology (Manicas and Secord) and education (Giroux and McUren). Indeed, it might 
be argued that the entire educational endeavor is currently undergoing those shifts that Kuhn 
describes. 

The prevailing conception of American higher education, unfortunately, still rests in the 
platonic notion of the Academy, that grove where teacher and disciples met in a purely intellectual 
atmosphere unrestrained by an administrative framework, As Peter M. Blau points out, however, H a 
romantic pleading for a return to a small community of scholars fails to take into account the 
demands made on higher education in modem society "(10). The shift from an agrarian to an 
industrial base, the .ise of literacy, the demands of a democratic society - all point to the simple fact 
that there is no going back to a pastoral, innocent time. If we reflect carefully, we will recaH that the 
past, a time in which the few benefited from the slavish labor of the many, was itself no glorious 
time. 

More specifically, recent national reports such as the much quoted National Institute of 
Education's Involvemen t in Learning: Realizing the Potential of American Higher Education point to 
the overwhelming need for a student oriented atmosphere. The authors write that: 

federal and state agencies, private foundations, colleges and universities, research 
organizational, and researchers concerned with higher education should focus 
their funding strategies and research activities on how to facilitate greater student 
learning and development. (74) 

Again, here is indiciated the need for a re-invention, a paradigm shift, for higher education. 
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In responding to these calls, E.T.S.U. administrators and faculty need not envision 
themselves about to embark on a Herculean labor. We should remember that 

E.T.S.U. is an institution known for its student-centerdness, 

E.T.S.U. already has in its possession a great deal of empirically formulated 

information about its students, 

E.T.S.U. is an educational leader in its region, and 

E.T.S.U. has always taken control in demanding situations. 

In essence, ET.S.U.'s administration end faculty already possess the criteria for success. 

In conclusion, then, the authors of the present work wish to call for discussion of methods to 
assess educational outcomes within our community. 
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11. A Reader's Guide to Research in Institutional Effectiveness 



The process of nongovernmental accreditation," Kerneth E. Young and Charles M. 
Chambers tell us, "is uniquely American " (90). It is the distinctly democratic nature of educational 
assessment that has prompted so much diverse scholarship over the past eighty years. Educational 
assessment is tied to the very nature of our society, and therefore the resulting research has been 
broad as well as deep. We do not exaggerate in stating that research in educational effectiveness as 
it relates to the process of accreditation is itself a discipline. 

Difficult as it is to find a point of entry, perhaps the best place to begin is with an 
understanding of the history of competency testing as set forth by David K. Cohen and Walter 
Haney. From that history readers might deepen their understanding by looking into Fred. F. 
Harcleroad's history of evaluation of educational programs in the U.S. since 1640. 

In order to contextualize our situation here in the south, we might then read in detail the 
Southern Association's Criteria for Accreditation and its Manual for Accreditation. With this specific 
background in mind, readers may turn to model programs of assessment described by Trudy W. 
Banta, Charles J. McClain and Darrell W. Krueger, and Marcia Mentkowski and Georgine Loacker. 

From here, readers may begin to venture forth through the monograph series sponsored by 
the Association for Institutional Research and published by Jossey-Bass. Currently, there are over 
fifty-six titles in this splendid series. In this series, a helpful system of bibliographical methodology is 
offered by Eugene C. Craven. 

In order to help those interested in research in institutional evaluation, the authors of the 
present work have compiled as a separate document a brief anthology of selected readings in the 
assessment of educational outcomes. We hope that the pieces contained there are both helpful and 
provocative. 
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